1. Field of the Invention
The present invention relates to the field of information retrieval and, more specifically, to measuring and improving search result relevance.
2. Description of the Prior Art
The relevance of a search result is the extent to which the result correlates to the intent of the user performing the search. Users often have a choice of systems on which to conduct a search, and such users are generally most likely to choose a system that provides them with the most relevant results. Thus, to attract and satisfy users, it is necessary to measure the relevance of results provided by a search system. Because result sets may often include a wide array of complex data, such relevance measurements often involve a detailed process of operations and calculations.
In conventional methods for measuring relevance, results are generally evaluated based on two factors: precision and recall. Precision is a measure of the purity of a result set or, more specifically, of how well a search avoids returning results that are not relevant. For example, if a search requests documents about “John Deere tractors”, and the search term “John Deere” is included in a submitted query, then the precision of the results would be lowered if the result set included documents that weren't about tractors but, rather, included references to a person whose name was “John Deer”.
Recall is a measure of the completeness of a result set. For example, if a search requests documents about “John Deere tractors”, then the recall of the results is raised if the result set includes documents about John Deere tractors, Ford tractors, etc. Thus, there is a trade-off between precision and recall. Generally, the higher the precision, the lower the recall, and, the higher the recall, the lower the precision.
A drawback of measuring relevance based on recall and precision is that they do not effectively measure user satisfaction, which is the extent to which the user perceives a result to correlate with his intent. There are several conditions which are likely to cause the user to be dissatisfied with highly precise and highly complete results. Such conditions may include, for example, poor content quantity, poor content quality, poor intent determination, poor result ranking, and poor result description.
In the case of poor content quality, the quality of the content that is being searched is insufficient to satisfy the user's intent. Even a highly precise and highly complete search of low quality content is unlikely to produce satisfactory results.
In the case of poor content quantity, the quantity of the content that is being searched is insufficient to satisfy the user's intent. Even a highly precise and highly complete search of low quantity content is unlikely to produce satisfactory results.
In the case of poor intent determination, the search system is unable to sufficiently determine the user's intent. Thus, if irrelevant content is being searched for, then even a highly precise and highly complete search is unlikely to produce satisfactory results.
In the case of poor result ranking, less relevant results are presented to a user before more relevant results. Such poor result ranking affects user satisfaction because users often consider a first few results rather than an entire set of results before determining their level of satisfaction with a search. Thus, poor result ranking may cause the user to quit a search or switch to another search system before encountering highly relevant results. Thus, if results are poorly ranked, then even a highly precise and highly complete search is unlikely to produce satisfactory results.
In the case of poor result description, the search system is unable to sufficiently describe results to the user. Users often base their decision of whether to request a result based on such a description. Thus, even if a result is highly relevant, the user may not request it if it is poorly described. Thus, if results are poorly described, then even a highly precise and highly complete search is unlikely to produce satisfactory results.
Because users ultimately determine on which system they will perform a search, user satisfaction is a highly important measure of relevance. Thus there is a need in the art for systems and methods for measuring and improving search result relevance based on user satisfaction.